V0.9.2 log trace #26283
V0.9.2 log trace #26283
Conversation
wanx7130
requested review from
NickLucche,
jikunshang,
bigPYJ1151,
mgoin,
tlrmchlsmth,
WoosukKwon,
yewentao256,
robertgshaw2-redhat,
njhill,
ywang96,
comaniac,
alexm-redhat,
heheda12345,
ApostaC and
LucasWilkinson
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
mergify
bot
added
documentation
|
This pull request has merge conflicts that must be resolved before it can be |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces significant changes to enable v0 execution on CPU, XPU, and TPU backends, including new worker implementations, model runners, and attention backends. It also includes refactoring of existing backends for better consistency and adds some debug logging. My review found a critical issue in the new CPU MLA backend implementation that would likely cause runtime errors.
| slot_mapping=slot_mapping, | ||
| multi_modal_placeholder_index_maps=placeholder_index_maps, | ||
| enable_kv_scales_calculation=False, | ||
| input_positions=torch.tensor([self.input_data.input_positions])) |
There was a problem hiding this comment.
The input_positions tensor is being created as a 2D tensor by wrapping self.input_data.input_positions in an extra list. The rotary embedding layer expects a 1D tensor for positions. This will likely lead to an indexing error or incorrect behavior at runtime.
| input_positions=torch.tensor([self.input_data.input_positions])) | |
| input_positions=torch.tensor(self.input_data.input_positions)) |
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
| if self.runner.attn_backend is not None: | ||
| # spec decode (e.g. Medusa) does not have atten backend | ||
| attn_backend = self.runner.attn_backend | ||
| self.att_metadata_builder = attn_backend.get_builder_cls()(self) | ||
|
|
||
| def prepare(self, | ||
| finished_requests_ids: Optional[List[str]] = None) -> None: | ||
| self.seq_group_metadata_list: List[SequenceGroupMetadata] = [] | ||
| self.input_data = ModelInputForCPUBuilder.ModelInputData( | ||
| self.runner.model_config.uses_mrope) | ||
| self.att_metadata_builder.prepare() |
There was a problem hiding this comment.
In ModelInputForCPUBuilder.prepare() the code unconditionally calls self.att_metadata_builder.prepare(), but self.att_metadata_builder is only created when self.runner.attn_backend is not None (see the comment about speculative decoding having no attention backend). When the runner is used for configurations that do not build an attention backend, prepare() will raise an AttributeError before any inputs are built, breaking CPU speculative decoding. Add a null check before invoking the builder or skip metadata preparation when no attention backend exists.
Useful? React with 👍 / 👎.
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.